osx_arm updates #1174

garrettwrong · 2024-09-13T11:17:12Z

Now that I have removed PyFFTW, pyshtools, and updated to FINUFFT 2.3.0 I think ASPIRE should be good to run native M1 out of the box. This PR patches the CI and removes the mention of the x86_64 conda install/workarounds.

I'll confirm with my own M1 machine a little later this morning before sending this for review.

garrettwrong · 2024-09-13T12:45:03Z

Had to patch a few more tests than I hoped, but overall I don't think we did too bad for a totally different architecture 😎 .

We may find a few more as it begins to run regularly, but they should mainly just be adding an appropriate tolerance (1e-6 seems to be working well for stuff; mainly seeing small diffs in singles).

Here is the run from my M1 mini:

(test_osx) ➜  ASPIRE-Python git:(osx_arm_update) ✗ git log | head
commit 8ae41ed233d371dd40a0c7796b24577f3c48d77a
Author: Garrett Wright <garrettwrong@gmail.com>
Date:   Fri Sep 13 08:33:59 2024 -0400

    Another test file tweak for arm64

commit 967f79cb44d87a5caf9afa5e9c152d45d3c4bf50
Author: Garrett Wright <garrettwrong@gmail.com>
Date:   Fri Sep 13 08:30:15 2024 -0400

(test_osx) ➜  ASPIRE-Python git:(osx_arm_update) ✗ time pytest -n10
====================================================================== test session starts ======================================================================
platform darwin -- Python 3.11.10, pytest-8.3.3, pluggy-1.5.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /Users/garrett/work/ASPIRE-Python
configfile: tox.ini
testpaths: tests
plugins: cov-5.0.0, random-order-1.1.1, anyio-4.4.0, xdist-3.6.1
10 workers [5664 items] kipped                                                                                                                                  
......................................................................................................................................................... [  2%]
.............................s.....ss.s.................................................................................................................. [  5%]
.......x..x.............................................................................................................................................. [  8%]
...................................................................................................................x.x................................... [ 10%]
.............................................xx...............xx.........................x..x.x.x...............................x...x.................... [ 13%]
.......x.x............................................................................x.x...xx....................................x.x......x.x........... [ 16%]
.....................xx..................................................................................................x..x............................ [ 18%]
.........................................................................................................................x...x........................... [ 21%]
.......................xx....................................................xxxx.........................x......x..........................x.x.......... [ 24%]
.........xx.xx......................xx........x...x.....................................................x.....................................x.......... [ 27%]
............................................................x.x.......................................xx................................................. [ 29%]
.......................................................................................................xx................................................ [ 32%]
......................................................................................................................................................... [ 35%]
......................................................................................................................................................... [ 37%]
......................................................................................................................................................... [ 40%]
......................................................................................................................................................... [ 43%]
......................................................................................................................................................... [ 45%]
......................................................................................................................................................... [ 48%]
......................................................................................................................................................... [ 51%]
...........................................................................................x............................................................. [ 54%]
......................................................................................................................................................... [ 56%]
..........................................................................................................X................................s............. [ 59%]
.......................s........s.....s.................................................................................................................. [ 62%]
...............x.x......................................xx......ssss..................xx........................xx....................................... [ 64%]
..........xx......................xx......................................................................................................xx............. [ 67%]
............xx........................xx..............................x....x...........................x.x...............................xx.............. [ 70%]
......................................................................................................................................................... [ 72%]
.........................................................x............................................................................................... [ 75%]
............................................................ssss......................................................................................... [ 78%]
......................................................................................................................................................... [ 81%]
......................................................................................................................................................... [ 83%]
....ss..ssss..ss......................................................................................................................................... [ 86%]
.....................................................................................................s.s........s...................................s.... [ 89%]
...s.s................................................................................................................................................... [ 91%]
......................................................................................................................................................... [ 94%]
......................................................................................................................................................... [ 97%]
......................................................................................................................................................... [ 99%]
...                                                                                                                                                       [100%]
======================================================================= warnings summary ========================================================================
tests/test_image.py: 1 warning
tests/test_micrograph_source.py: 8 warnings
tests/test_simulation.py: 3 warnings
  /Users/garrett/work/ASPIRE-Python/src/aspire/image/image.py:691: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
    plt.show()

tests/test_image.py: 1 warning
tests/test_fourier_correlation.py: 32 warnings
tests/test_micrograph_source.py: 8 warnings
tests/test_simulation.py: 3 warnings
  /Users/garrett/work/ASPIRE-Python/tests/test_utils.py:403: MatplotlibDeprecationWarning: Auto-close()ing of figures upon backend switching is deprecated since 3.8 and will be removed in 3.10.  To suppress this warning, explicitly call plt.close('all') first.
    matplotlib.use(backend)

tests/test_fourier_correlation.py: 32 warnings
  /Users/garrett/work/ASPIRE-Python/src/aspire/utils/resolution_estimation.py:386: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
    plt.show()

tests/test_fourier_correlation.py::test_frc_img_plot[img_size=64-dtype=<class 'numpy.float64'>]
tests/test_fourier_correlation.py::test_frc_img_plot[img_size=64-dtype=<class 'numpy.float32'>]
tests/test_fourier_correlation.py::test_frc_img_plot[img_size=65-dtype=<class 'numpy.float64'>]
tests/test_fourier_correlation.py::test_frc_img_plot[img_size=65-dtype=<class 'numpy.float32'>]
tests/test_fourier_correlation.py::test_fsc_vol_plot[img_size=64-dtype=<class 'numpy.float64'>]
tests/test_fourier_correlation.py::test_fsc_vol_plot[img_size=64-dtype=<class 'numpy.float32'>]
tests/test_fourier_correlation.py::test_fsc_vol_plot[img_size=65-dtype=<class 'numpy.float64'>]
tests/test_fourier_correlation.py::test_fsc_vol_plot[img_size=65-dtype=<class 'numpy.float32'>]
  /Users/garrett/work/ASPIRE-Python/src/aspire/utils/resolution_estimation.py:380: UserWarning: No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
    plt.legend(title=f"Method: {self.method}")

tests/test_micrograph_source.py::test_rectangular_micrograph_source_files
  /Users/garrett/miniconda3/envs/test_osx/lib/python3.11/site-packages/mrcfile/mrcobject.py:626: RuntimeWarning: Data array contains NaN values
    warnings.warn("Data array contains NaN values", RuntimeWarning)

tests/test_covar2d_denoiser.py: 15 warnings
  /Users/garrett/work/ASPIRE-Python/src/aspire/basis/steerable.py:524: UserWarning: BlkDiagMatrix.from_dense truncating values exceeding 0.001
    filt = BlkDiagMatrix.from_dense(

tests/test_sinogram.py: 60 warnings
  /Users/garrett/work/ASPIRE-Python/src/aspire/image/image.py:243: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
    return aspire.sinogram.Sinogram(xp.asnumpy(image_rt))

tests/test_sinogram.py: 20 warnings
  /Users/garrett/work/ASPIRE-Python/src/aspire/sinogram/sinogram.py:62: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
    return self.__class__(self._data[key])

tests/test_sinogram.py: 22 warnings
  /Users/garrett/work/ASPIRE-Python/src/aspire/sinogram/sinogram.py:90: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
    return self.__class__(self._data.reshape(*shape, *self._data.shape[-2:]))

tests/test_sinogram.py: 12 warnings
  /Users/garrett/miniconda3/envs/test_osx/lib/python3.11/site-packages/skimage/transform/radon_transform.py:74: UserWarning: Radon transform: image must be zero outside the reconstruction circle
    warn(

tests/test_commonline_sync3n.py::test_estimate_rotations[resolution=40-offsets=0-dtype=<class 'numpy.float32'>]
  /Users/garrett/work/ASPIRE-Python/src/aspire/abinitio/commonline_sync3n.py:730: RuntimeWarning: overflow encountered in exp
    Pij = 1 / (1 + (1 - P) / P * np.exp(ln_f_arb - ln_f_ind))

tests/test_orient_symmetric.py: 1932 warnings
  /Users/garrett/work/ASPIRE-Python/tests/test_orient_symmetric.py:158: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    dist[s] = np.minimum(

tests/test_orient_symmetric.py: 192 warnings
  /Users/garrett/work/ASPIRE-Python/tests/test_orient_symmetric.py:195: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    dist[i] = Rotation.angle_dist(estimate, Rii_gt)

tests/test_orient_symmetric.py: 2970 warnings
  /Users/garrett/work/ASPIRE-Python/tests/test_orient_symmetric.py:641: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    clmatrix_gt[idx, i, j] = CLSymmetryC3C4.cl_angles_to_ind(

tests/test_orient_symmetric.py: 2970 warnings
  /Users/garrett/work/ASPIRE-Python/tests/test_orient_symmetric.py:644: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    clmatrix_gt[idx, j, i] = CLSymmetryC3C4.cl_angles_to_ind(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================== 5551 passed, 32 skipped, 82 xfailed, 1 xpassed, 8291 warnings in 185.03s (0:03:05) =======================================
pytest -n10  936.99s user 137.56s system 579% cpu 3:05.40 total

j-c-c

Looks good! Just one comment.

tests/test_FLEbasis2D.py

j-c-c

Looks good!

tests/test_FLEbasis2D.py

garrettwrong · 2024-09-13T17:46:59Z

No functional changes, just the CI and some testing tolerances. Merging.

garrettwrong added documentation Improvements or additions to documentation enhancement New feature or request CI Continuous Integration cleanup extern Relating to external changes dependencies Pull requests that update a dependency file labels Sep 13, 2024

garrettwrong self-assigned this Sep 13, 2024

garrettwrong force-pushed the osx_arm_update branch 2 times, most recently from f870123 to 8ae41ed Compare September 13, 2024 12:44

garrettwrong requested a review from j-c-c September 13, 2024 12:45

j-c-c previously approved these changes Sep 13, 2024

View reviewed changes

tests/test_FLEbasis2D.py Outdated Show resolved Hide resolved

garrettwrong dismissed j-c-c’s stale review via 904df59 September 13, 2024 13:43

garrettwrong requested a review from j-c-c September 13, 2024 13:46

j-c-c approved these changes Sep 13, 2024

View reviewed changes

tests/test_FLEbasis2D.py Show resolved Hide resolved

garrettwrong marked this pull request as ready for review September 13, 2024 14:32

garrettwrong requested a review from janden as a code owner September 13, 2024 14:32

garrettwrong added 9 commits September 13, 2024 12:08

migrate osx_arm CI job to run native python/pip

085da47

Remove x86_64 doc updates

9d13bf7

Use 3.11 on osx_arm, default was 3.12

63e3639

use default osx shell

1713172

adjust a few tests for osx arm

01f5744

convert simulation tests to np.testing.assert_allclose

29de265

Another test file tweak for arm64

bfc0c30

loosen FLE for arm a little more

8a347a0

adjust comment

1734261

garrettwrong force-pushed the osx_arm_update branch from 091e5b6 to 1734261 Compare September 13, 2024 16:08

garrettwrong merged commit c094e56 into develop Sep 13, 2024

garrettwrong deleted the osx_arm_update branch September 13, 2024 17:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

osx_arm updates #1174

osx_arm updates #1174

Uh oh!

garrettwrong commented Sep 13, 2024

Uh oh!

garrettwrong commented Sep 13, 2024

Uh oh!

j-c-c left a comment

Uh oh!

Uh oh!

j-c-c left a comment

Uh oh!

Uh oh!

garrettwrong commented Sep 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

osx_arm updates #1174

osx_arm updates #1174

Uh oh!

Conversation

garrettwrong commented Sep 13, 2024

Uh oh!

garrettwrong commented Sep 13, 2024

Uh oh!

j-c-c left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

j-c-c left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

garrettwrong commented Sep 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants